Secure Lossy Transmission of Vector Gaussian Sources* 



Ersen Ekrem 



Sennur Ulukus 



Department of Electrical and Computer Engineering 
University of Maryland, College Park, MD 20742 



ersen@umd.edu 



ulukus @umd. edu 



August 18, 2011 



Abstract 



We study the secure lossy transmission of a vector Gaussian source to a legiti- 
mate user in the presence of an eavesdropper, where both the legitimate user and the 
eavesdropper have vector Gaussian side information. The aim of the transmitter is 
to describe the source to the legitimate user in a way that the legitimate user can 
reconstruct the source within a certain distortion level while the eavesdropper is kept 
ignorant of the source as much as possible as measured by the equivocation. We ob- 
tain an outer bound for the rate, equivocation and distortion region of this secure lossy 
transmission problem. This outer bound is tight when the transmission rate constraint 
is removed. In other words, we obtain the maximum equivocation at the eavesdrop- 
per when the legitimate user needs to reconstruct the source within a fixed distortion 
level while there is no constraint on the transmission rate. This characterization of the 
maximum equivocation involves two auxiliary random variables. We show that a non- 
trivial selection for both random variables may be necessary in general. The necessity 
of two auxiliary random variables also implies that, in general, Wyner-Ziv coding is 
suboptimal in the presence of an eavesdropper. In addition, we show that, even when 
there is no rate constraint on the legitimate link, uncoded transmission (deterministic 
or stochastic) is suboptimal; the presence of an eavesdropper necessitates the use of a 
coded scheme to attain the maximum equivocation. 



*This work was supported by NSF Grants CCF 07-29127, CNS 09-64632, CCF 09-64645 and CCF 10- 
18185. 
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1 Introduction 



Information theoretic secrecy was initiated by Wyner in [TJ, where he studied the secure 
lossless transmission of a source over a degraded wiretap channel, and obtained the necessary 
and sufficient conditions. Later, his result was generalized to arbitrary, i.e., not necessarily 
degraded, wiretap channels in [2] . In recent years, information theoretic secrecy has gathered 
a renewed interest, where mostly channel coding aspects of secure transmission is considered, 
in other words, secure transmission of uniformly distributed messages is studied. 

Secure source coding problem has been studied for both lossless and lossy reconstruction 
cases in [3HT6]. Secure lossless source coding problem is studied in [3H9]. The common theme 
of these works is that the legitimate receiver wants to reconstruct the source in a lossless 
fashion by using the information it gets from the transmitter in conjunction with its side 
information, while the eavesdropper is being kept ignorant of the source as much as possible. 
Secure lossy source coding problem is studied in [TOHTB] . In these works, unlike the ones 
focusing on secure lossless source coding, the legitimate receiver does not want to reconstruct 
the source in a lossless fashion, but within a distortion level. 

The most relevant works to our work here are p3HT6] . In |15j . the author considers the se- 
cure lossy transmission of a source over a degraded wiretap channel while both the legitimate 
receiver and the eavesdropper have side information about the source. In [15] , in addition 
to the degradedness that the wiretap channel exhibits, the source and side information also 
have a degradedness structure such that given the legitimate user's side information, the 
source and the eavesdropper's side information are independent. For this setting, in [15], a 
single-letter characterization of the distortion and equivocation region is provided. In par- 
ticular, the optimality of a separation-based approach, i.e., the optimality of a code that 
concatenates a rate-distortion code and a wiretap channel code, is shown. In [16], the set- 
ting of [15J is partially generalized such that in [16], the source and side information do 
not have any degradedness structure. On the other hand, as opposed to the noisy wiretap 
channel of [15], in [16], the channel between the transmitter and receivers is assumed to be 
noiseless. For this setting, in [16], a single- letter characterization of the rate, equivocation, 
and distortion region is provided. 

Here, we consider the setting of [16] for jointly Gaussian source and side information. In 
particular, we consider the model where the transmitter has a vector Gaussian source which 
is jointly Gaussian with the vector Gaussian side information of both the legitimate receiver 
and the eavesdropper. In this model, the transmitter wants to convey information to the 
legitimate user in a way that the legitimate user can reconstruct the source within a distor- 
tion level while the eavesdropper is being kept ignorant of the source as much as possible 
as measured by the equivocation. A single-letter characterization of the rate, equivocation, 
and distortion region for this setting exists due to [16]. Although we are unable to evaluate 
this single-letter characterization for the vector Gaussian source and side information case to 
obtain the corresponding rate, equivocation, distortion region explicitly, we obtain an outer 
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bound for this region. We obtain this outer bound by optimizing the rate and equivocation 
constraints separately. We note that a joint optimization of the rate and equivocation con- 
straints for a fixed distortion level would yield the exact achievable rate and equivocation 
region for this fixed distortion level. Thus, optimizing the rate and equivocation constraints 
separately yields a larger region, i.e., an outer bound. We show that this outer bound is 
tight when we remove the rate constraint at the transmitter. In other words, we obtain the 
maximum achievable equivocation at the eavesdropper when the legitimate user needs to 
reconstruct the vector Gaussian source within a fixed distortion while there is no constraint 
on the transmission rate. 

We note some implications of this result. First, we note that since there is no rate 
constraint on the transmitter, it can use an uncoded scheme to describe the source to the 
legitimate user, and, indeed, it can use any instantaneous (deterministic or stochastic) en- 
coding scheme for this purpose. However, we show through an example that even when 
there is no rate constraint on the transmitter, to attain the maximum equivocation at the 
eavesdropper, in general, the transmitter needs to use a coded scheme. Hence, the presence 
of an eavesdropper necessitates the use of a coded scheme even in the absence of a rate 
constraint on the transmitter. Second, we note that the maximum equivocation expression 
has two different covariance matrices originating from the presence of two auxiliary random 
variables in the single-letter expression. We show through another example that both of 
these covariance matrices, in other words, both of these two auxiliary random variables, are 
needed in general to attain the maximum equivocation at the eavesdropper. The necessity of 
two covariance matrices, and hence two auxiliary random variables, implies that, in general, 
Wyner-Ziv coding scheme [17] is not sufficient to attain the maximum equivocation at the 
eavesdropper. 

2 Secure Lossy Source Coding 

Here, we describe the secure lossy source coding problem and state the existing results. Let 
{(Xi,Yi, Zi)}f =1 denote i.i.d. tuples drawn from a distribution p(x,y,z). The transmitter, 
the legitimate user and the eavesdropper observe X n e X n , Y n G y n , and Z n e Z n , respec- 
tively. The transmitter wants to convey information to the legitimate user in a way that 
the legitimate user can reconstruct the source X n within a certain distortion, and mean- 
while the eavesdropper is kept ignorant of the source X n as much as possible as measured 
by the equivocation. We note that if there was no eavesdropper, this setting would reduce 
to the Wyner-Ziv problem [T7J, for which a single-letter characterization for the minimum 
transmission rate of the transmitter for each distortion level exists. 

The distortion of the reconstructed sequence at the legitimate user is measured by the 
function d n (X n , X n ) where X n e X n denotes the legitimate user's reconstruction of the 
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source X n . We consider the function d n (X n ,X n ) that has the following form 

1 - 

( r(x n ,x n ) = ~Y t d(x i ,x i ) (i) 

i=i 

where d(a, b) is a non-negative finite-valued function. The confusion of the eavesdropper is 
measured by the following equivocation term 

-H(X n \Z n ,M) (2) 
n 

where M G M, which is a function of the source X n , denotes the signal sent by the trans- 
mitter. 

An (n, R) code for secure lossy source coding consists of an encoding function f n : X n — > 
= {1, . . . , 2 nR } at the transmitter and a decoding function at the legitimate user g n : 

A4 x y n — > X n . A rate, equivocation and distortion tuple (R, R e , D) is achievable if there 

exists an (n, R) code satisfying 

lim -H(X n \Z n ,M) > R e (3) 

n-voo n 

lim E[d(X n ,X n )] < D (4) 

n— too 

The set of all achievable (R, R e , D) tuples is denoted by 1Z* which is given by the following 
theorem. 

Theorem 1 ([TBI Theorem 1]) (R, R e , D) e 11* iff 



R>I(V;X\Y) (5) 

R e < H(X\V, Y) + I(X; Y\U) - I(X; Z\U) (6) 

D>E[d(X,X{V,Y))} (7) 

for some U, V satisfying the following Markov chain 

U^V^X^Y,Z (8) 



and a function X(V, Y). 

The achievable scheme that attains the region 1Z* has the same spirit as the Wyner-Ziv 
scheme [17] in the sense that both achievable schemes use binning to exploit the side informa- 
tion at the legitimate user, and consequently, to reduce the rate requirement. The difference 
of the achievable scheme that attains TZ* comes from the additional binning necessitated by 
the presence of an eavesdropper. In particular, the transmitter generates sequences (U n , V n ) 
and bins both sequences. The transmitter sends these two bin indices. Using these bin in- 
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dices, the legitimate user identifies the right (U n , V n ) sequences, and reconstructs X n within 
the required distortion. On the other hand, using the bin indices of (U n ,V n ), the eaves- 
dropper identifies only the right U n sequence, and consequently, U does not contribute to 
the equivocation, see «fl Indeed, this achievable scheme can be viewed as if it is using a 
rate-splitting technique to send the message M, since M has two coordinates, one for the 
bin index of U n , and one for the bin index of V n . This perspective reveals the similarity of 
the achievable scheme that attains 1Z* and the one that attains the capacity-equivocation 
region of the wiretap channel [2] where also rate-splitting is used. In particular, in the latter 
case, the message W is divided into two parts W ne , W e such that W ne is sent by the sequence 
U n and W e is sent by the sequence V n . The eavesdropper decodes W ne whereas the other 
message W e contributes to the secrecy. 

We note that Theorem [1] holds for continuous (X n , Y n , Z n ) by replacing the discrete 
entropy term H(X\V,Y) with the differential entropy term h(X\V,Y). To avoid the neg- 
ative equivocation that might arise because of the use of differential entropy, we replace 
equivocation with the mutual information leakage to the eavesdropper I e defined by 

lim -I{X n ;Z n ,M) (9) 

n— s-oo Ti 

Once we are interested in the mutual information leakage to the eavesdropper, a rate, mutual 
information leakage, and distortion (R, I e , D) tuple is said to be achievable if there exists an 
(n, R) code such that 

lim -I{X n ;Z n ,M) < I e (10) 
lim E[d(X n ,X n )] < D (11) 

n— voo 

The set of all achievable (R,I e ,D) tuples is denoted by 1Z. Using Theorem [U the region 71 
can be stated as follows. 

Theorem 2 (pU) (R, J e , D) E TZ iff 



R>I{V;X\Y) (12) 

h > I(V; X) - IiV- Y\U) + I(X; Z\U) (13) 

D>E[d{X,X{V,Y))} (14) 

for some U, V satisfying the following Markov chain 

U^V^X^Y,Z (15) 



1 The fact that the eavesdropper can decode U n sequence can be obtained by observing that for a (U, V) 
selection, if I(U ; Y) > I(U; Z), there is no loss of optimality of setting U = 4> which will yield a larger region. 
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and a function X(V, Y). 



3 Vector Gaussian Sources 

Now we study the secure lossy source coding problem for jointly Gaussian {(Xj, Yj, Zj)}" =1 
where the tuples {(Xj, Yj, Zj)}" =1 are independent across time, i.e., across the index z, and 
each tuple is drawn from the same jointly Gaussian distribution p(X, Y, Z). In other words, 
we consider the case where Xj is a zero-mean Gaussian random vector with covariance matrix 
>~ 0, and the side information at the legitimate user Y, and the eavesdropper Zj are 
jointly Gaussian with the source Xj. In particular, we assume that Yj, Zj have the following 
form 

Y t = X, + N Y>i (16) 
Z, = Xj + N Z)< (17) 

where Nyj and "Nz,i are independent zero-mean Gaussian random vectors with covariance 
matrices Sy y and Hz >- 0, respectively, and (Nyj, N Zj j) and Xj are independent. We 
note that the side information given by (116j) -( ll7p are not in the most general form. In the 
most general case, we have 

Yj = HyXj + Ny ( j (18) 

Z, = H Z X, + N Z)< (19) 

for some Hy, Hz matrices. However, until Section EJ we consider the form of side information 
given by (|T6!) - (|T7j) . and obtain our results for this model. In Section [5], we generalize our 
results to the most general case given by (|T8|) - (|T9|) . We note that since the rate, information 
leakage and distortion region is invariant with respect to the correlation between Ny t j and 
~Nz,i, the correlation between Nyj and N Z) j is immaterial. 

The distortion of the reconstructed sequence {Xj}" =1 is measured by the mean square 
error matrix: 



E 



(Xj-X,)(X,-X t ) T j (20) 



Hence, the distortion constraint is represented by a positive semi-definite matrix D, which 
is achievable if there is an (n, R) code such that 

-^£[(X,-X t )(X,-X 4 ) T ] (21) 

i=l 

Throughout the paper, we assume that ^ D ^ Kx|y. Since the mean square error is 
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minimized by the minimum mean square error (MMSE) estimator which is given by the 
conditional mean, we assume that the legitimate user applies this optimal estimator, i.e., 
the legitimate user selects its reconstruction function {Xj}™ =1 as 

X i = J E7[X i |Y n ,/ n (X n )] (22) 

Once the estimator of the legitimate user is set as (I22j) . using Theorem El a single-letter 
description of the region 1Z for a vector Gaussian source can be given as follows. 

Theorem 3 (R, I e , D) 6 K iff 

R>I{V-X.\Y) (23) 
I e > I(V; X) - I(V; Y\U) + /(X; Z\U) (24) 
D h K X{VY (25) 

for some U, V satisfying the following Markov chain 

t/^V^X^Y,Z (26) 

We also define the region TZ(D) as the union of the (R, I e ) pairs that are achievable when the 
distortion constraint matrix is set to D. Our main result is an outer bound for the region 
7£(D), hence for the region TZ. 

Theorem 4 When D ^ K^ry, we have 

n(D) C TZ°(D) (27) 
where 1Z°(D) is given by the union of (R, I e ) that satisfy 

R > t: log — — = - log — — - - log ————— (28) 



2 D 2 FD 2 F(D) + S 



U > mm -log-— r log ^ r =— r + - log ^— 29 

K X]V ^F(D) 

and F(D) = S y (S y - D)- 1 S y - Sy. 

We will prove Theorem H] in Section HI In the remainder of this section, we provide interpre- 
tations and discuss some implications of Theorem HI 

The outer bound in Theorem H] is obtained by minimizing the constraints on R and I e 
individually, i.e., the rate lower bound in ( 1281) is obtained by minimizing the rate constraint 
in (|23|) and the mutual information leakage lower bound in ( )29|) is obtained by minimizing 
the mutual information leakage constraint in (|24|) separately. However, to characterize the 
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rate and mutual information leakage region 1Z(D), one needs to minimize the rate constraint 
in (1231) and the mutual leakage information constraint in (124")) jointly, not separately. In 
particular, since the region 7Z(D) is convex in the pairs (R, I e ) as per a time-sharing argu- 
ment, joint optimization of the rate constraint in ( 1231) and the mutual information leakage 
constraint in fl24j) can be carried out by considering the tangent lines to the region 1Z(D), 
i.e., by solving the following optimization problem 

£(^1,^2) = min + /z 2 / e (30) 

(R,ie)en(r>) 

= min ui [I(V; X) - J(V; Y)] + ^ [I(V; X) - I(V; Y\U) + /(X; Z\U)) 

[/— >V— >X->Y.Z 

(31) 

for all values of /ii,/i2, where /ij G [0, 00), j = 1,2. As of now, we have been unable 
to solve the optimization problem L(/xi,/i 2 ) for all values of (/xi,/i 2 ). However, as stated 
in Theorem HI we solve the optimization problems L(0,/i 2 ) and L(/i 1; 0) by showing the 
optimality of jointly Gaussian (U, V, X) to evaluate the corresponding cost functions. In 
other words, our outer bound in Theorem @] can be written as follows. 

R>L(1,0) (32) 
I e > L(0, 1) (33) 

We note that the constraint in fl28|) . and hence L(l, 0), gives us the Wyner-Ziv rate distortion 
function [T7j for the vector Gaussian sources. Moreover, we note that L(0, 1) gives us the 
minimum mutual information leakage to the eavesdropper when the legitimate user wants to 
reconstruct the source within a fixed distortion constraint D while there is no concern on the 
transmission rate R. Denoting the minimum mutual information leakage to the eavesdropper 
when the legitimate user needs to reconstruct the source within a fixed distortion constraint 
D by /™ m (D), the corresponding result can be stated as follows. 

Theorem 5 When D < Kxiy, we have 



, /-r^ • 1, |Kx| 1, |K X |(7 + £y| 1 \Kx\u + ^z\ , OA , 
II (D) = mm -log- 1 — log-; + - log : 34) 

' 1 ; o^k x|v ^k x|[7 ^k x 2 \K xlv \ 2 B \K xiv + 1:y\ 2 S |S Z | V ; 



" K X[V ^F(D) 



where F(D) = S y (S y - D)" 1 S y - S y . 

Theorem [5] implies that if the transmitter's aim is to minimize the mutual information 
leakage to the eavesdropper without concerning itself with the rate it costs as long as the 
legitimate receiver is able to reconstruct the source within a distortion constraint D, the use 
of jointly Gaussian (U, V, X) is optimal. Since in Theorem [5], there is no rate constraint, 
one natural question to ask is whether 7™ m (D) can be achieved by an uncoded transmission 
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scheme. Now, we address this question in a broader context by letting the encoder use any 
instantaneous encoding function in the form of <7i(Xj) where gi(-) can be a deterministic or 
a stochastic mapping. When is chosen to be stochastic, we assume it to be independent 
across time. We note that the uncoded transmission can be obtained from instantaneous 
encoding by selecting g^-) to be a linear function. Similarly, uncoded transmission with 
artificial noise can be obtained from instantaneous encoding by selecting gi{x) = ax + N, 
where N denotes the noise. Hence, if the encoder uses an instantaneous encoding scheme, 
the transmitted signal is given by M = [ pi(Xi), . . . , g> n (X„) ]. Let Jg ns (D) be the minimum 
information leakage to the eavesdropper when the legitimate user is able to reconstruct the 
source with a distortion constraint D while the encoder uses an instantaneous encoding. The 
following example demonstrates that, in general, J™ m (D) cannot be achieved by instanta- 
neous encoding. 

Example 1 Consider the scalar case, where the side information at the legitimate user and 
the eavesdropper are given as follows 

Y i = X i + Ny 7 i (35) 
Zi = Xi + N Z} i (36) 

where Xi,N Vj i and N Zt i are zero-mean Gaussian random variables with variances 0^,0^ and 
a 2 z , respectively. {Xj}" =1 , {N y> i}f =1 and {N Zti }™ =1 are independent. We assume that a 2 . < a 2 z , 
which implies that we can assume X — » Y — >■ Z since the scalar model in l[35\) - l[3o)) is 
statistically degraded, or in other words, the correlation between N y ^ and N Z)i does not affect 
the achievable (R,I e ,D) region. Using Theorem^ I™ m (D) for the scalar Gaussian channel 
under consideration can be found as follows 

lf n (D)= min I(V;X)-I(V;Y\U) + I(X;Z\U) (37) 

U^V^X->Y^Z 
a 2 , <D 

x \vy — 

= min I(V;X) - I(V;Y) + I(X;Z) (38) 

a 2 , <D 

x\vy — 

where in we used the Markov chain U^-V-^X^-Y^-Z. 

As shown in Appendix\A\ the information leakage to the eavesdropper when the encoder 
uses an instantaneous mapping is given by 

I^ S (D)= min I(X;V,Z) (39) 

a 2 , <D 

x \vy — 

min I(V; X) - I(V; Z) + I(X; Z) (40) 

a 2 , <D 

x \vy — 

where ( f^Op is obtained by using the Markov chain V — > X — > Z. 
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Using UJIfy and UW , we have 



I™{D)-ir{D)= min I(V ; X) - I(V ; Z) + I(X; Z) 

V^tX^Y^rZ 

a 2 , <D 

x \vy — 

- min I(V;X) - I(V;Y) + I(X;Z) (41) 

V^X^Y^Z 
a 2 , <D 

x \ vy — 

> min I(V ; Y) - I(V; Z) (42) 

V^X^Y^Z 
a 2 , <D 

x \ vy — 

min I(V:Y\Z) (43) 
v->x->y->z 

a 2 , <D 

x \vy — 

where f^3| ) comes from the Markov chain V — >• Y — )■ Z . iVerrf, we note £/ie following lemma. 



Lemma 1 For jointly Gaussian (X, Y, Z) satisfying the Markov chain X — > Y — > Z and 
Pr[F = Z] ^ I, ifD < a 2 x{y , we have 

min I(V:Y\Z)>0 (44) 

V^X^Y^Z 
a 2 , <D 

x \vy — 

The proof of Lemma Ul can be found in Appendix O The proof of Lemma U\ starts with the 
observation that f^| ) is zero iff we have the Markov chain V — )■ Z — > Y '. On the other hand, 
since we already have the Markov chain V — > X — > Y — > Z , and Y and Z are not identical, 
we show in Appendix [5| that the Markov chain V — >■ Z — > Y is possible iff V and X are 
independent. However, if D < txV, any V that is independent of X is not feasible. Hence, 
LemmaUl follows. LemmaUl implies that in general, we have /g ns (.D) ^ I™ n (D), i.e., J™ m (D) 
cannot be achieved by instantaneous encoding. 

This example shows that an uncoded transmission is not optimal even when there is no rate 
constraint. This is due to the presence of an eavesdropper; the presence of an eavesdropper 
necessitates the use of a coded scheme. 

Another question that Theorem [5] brings about is whether the minimum in (|34|) is achieved 
by a non-trivial K~x\u- By a trivial selection for K.x\u we mean either K.x\u — Kj or 
Kx|(j = Kx|v- The former corresponds to the selection U = <fi and the latter corresponds 
to the selection U — V. We note that although is monotonically decreasing in Kx|y in 
the positive semi-definite sense, (1341) is neither monotonically increasing nor monotonically 
decreasing in K~x\u i n the positive semi-definite sense. Hence, due to this lack of monotonicity 
of fl34"|) in ~K.x\Ui m general, we expect that both U ^ <p an d U ^ V may be necessary to 
attain the minimum in ( l34l) . The following example demonstrates that in general U ^ <f> and 
U 7^ V may be necessary. 
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Example 2 Consider the Gaussian source X = [ X\ X 2 ] T where X\ and X2 are indepen- 
dent. The side information at the legitimate receiver and the eavesdropper are given by 

Y e = X e + N Ytt , £=1,2 (45) 
Z e = X e + N z/ , £ = 1,2 (46) 

where Ny,e and Nzi are zero-mean Gaussian random variables with variances a Y £ and o~\ t , 
respectively. Moreover, N Y) i and iVy,2 are independent, and also so are N z ,i and N z>2 . We 
assume that noise variances satisfy 

4.1 < 4,i (47) 

4.2 < 4,2 (48) 

which, in view of the fact that correlation between the noise at the legitimate receiver and the 
noise at the eavesdropper does not affect the rate, distortion and information leakage region, 
lets us assume the following Markov chains 



Xi Yi Z x (49) 
X 2 -> Z 2 ->• Y 2 (50) 



Moreover, we assume that the distortion constraint D is a diagonal matrix with diagonal 
entries D\ and D 2 . In this case, the minimum information leakage is given by 

ir(D 1 ,D 2 )= min I(Vi; Xi) — I(Vi; Yi) + I(X 1 ; Z\) 

° 2 Xll v lYl <Di 

+ min I(V 2 ;X 2 ) + I(X 2 ;Z 2 \V 2 ) (51) 

V 2 ^-X 2 ^Z 2 ^-Y 2 

°X 2 \V 2 Y 2 < D 2 

whose proof can be found in Appendix^ The minimum information leakage in ( f5lj) corre- 
sponds the selections U = ((f>, V 2 ) and V = (Vi, V 2 ), where (U\, v\) and (U 2 , V 2 ) are indepen- 
dent. This selection of (U, V) corresponds to neither U = <fi nor U = V . 

Next, we obtain the minimum information leakage that arises when we set either U = <fi 
or U = V , and show that the minimum information leakage arising from these selections 
are strictly larger than the minimum information leakage in ( f5lj) . which will imply the sub- 
optimality of ' U = and U = V . When we set U = <f>, the minimum information leakage is 
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given by 



l?*-+(D 1 ,D 2 )= min I{V x] - J(V i; Y t ) + J(X l5 Z x ) 

Vi-*Xi-*Y\-+Z\ 
a 2 XlWlYl <Di 

+ min I{V 2 ; X 2 ) - I(V 2 ; Y 2 ) + I{X 2 ; Z 2 ) (52) 

V2^X 2 ^Z 2 ^Y 2 

*X 2 \V 2 Y 2 < D 2 

whose proof is given in AppendixlDi When we set U = V, the minimum information leakage 
is given by 

ir~ s (D 1 ,D 2 )= min J(V i; X x ) + I(X X - Z 1 \V 1 ) 

Vi-+Xi->Yi-±Zi 
a% l[VlYl <Di 

+ min I(V 2 ;X 2 )+I(X 2 ;Z 2 \V 2 ) (53) 

Vz-tXn-tZa-tYa 

°X 2 \V 2 Y 2 < D * 

whose proof can be found in Appendix [Dl 

Now, we compare the minimum information leakage in (51\) with ( f5l?j) and ( f53j) to s/iow 
t/iat i/ie selections U = <fi and U = V are sub-optimal in general. Using 137]) and / f5^j) . we 

/mi n^ (jDi;jD2) _ /m in (jDi;jD2)= ^ J(\/ 2 ;X 2 )-/(V r 2 ;r 2 ) + /(X 2 ;Z 2 ) 

v 2 ->-x 2 ->-z 2 ->-y 2 

°X 2 \V 2 Y 2 < D 2 

min J(\/ 2 ;X 2 ) + /(X 2 ;Z 2 |V 2 ) (54) 

> min /(X 2 ;Z 2 ) -J(X 2 ;Z 2 |V 2 ) - I{V 2 ;Y 2 ) (55) 

4 2 |V 2i - 2 < D 2 

= min J(V a ;Z 2 )-I(V 3 ;Y a ) (56) 

v 2 ->-x 2 -)-z 2 ->-y 2 

4 2 |v 2 y 2 < D 2 

= min I(V 2 ;Z a |y a ) (57) 

<4 2 |v 2 y 2 < D 2 

> (58) 
where [56]) - (51) follow from the Markov chain 

V 2 ^X 2 ^Z 2 ^ Y 2 (59) 

and (E2[) comes from LemmaUl Thus, in general, we have I^ m ~^(Di, D 2 ) ^ I™ m (Di, D 2 ) , 
or in other words, in general, U = is sub-optimal. 
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Next, we consider the selection U = V. Using l[51\) and (G3|) ; we have 



I?*-*{D X ,D 2 )-I?*(D 1 ,D*) = min I{V 1 -X l ) + I{X l] Z 1 \V 1 ) 

°% llVlYl <Di 

- min I(V 1] X 1 )-I(V 1] Y 1 )+I(X 1] Z 1 ) (60) 

Vi—tXi— >Yi— >Zi 
<? 2 x llVlYl <Di 

> min /(Xi^xlFO+J^;^- I{X X ;Z X ) (61) 

Vi->Xi-»Yi->Zi 

= min /(V^yO-Z^uZx) (62) 

Vi->Xi->Yi->2i 

= min /(Vu^lZO (63) 
Vi— tX\-*Y\-*Zi 

> (64) 
where ([H|)-([22P follow from the Markov chain 

V l ^X 1 ^Y 1 ^ Z x (65) 

and ( pm ) comes from LemmaUi Thus, in general, we have I™ m (D x , D 2 ) 7^ /™ m (Z)i, -D2); 
or m ot/ier words, in general, U = V is sub-optimal. 

Example [2] shows that, in general, we might need two covariance matrices, and hence two 
different auxiliary random variables, to attain the minimum information leakage. Indeed, if 
we have either U = V or U = <p, the corresponding achievable scheme is identical to the 
Wyner-Ziv scheme [17J. Hence, the necessity of two different auxiliary random variables 
implies that, in general, Wyner-Ziv scheme p2] is suboptimal. 



4 Proof of Theorem [4 

We now provide the proof of Theorem HI As mentioned in the previous section, this outer 
bound is obtained by minimizing the rate constraint in ( 123]) and the mutual information 
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leakage constraint in ( 124)) separately. We first consider the rate constraint in (|23|) as follows 



R>L(1,0) (66) 
= min J(V;XIY) (67) 

V->X^Y,Z 

= min h(X\Y) - h(X\V,Y) (68) 

K->X->Y,Z 

= min ^log |(27re)K X | Y | — h(X\V,Y) (69) 
k x]vy ±t> 

1 l l K *H 

mm - log — r (70) 



- , l K *l 

- log — n— 

2 D 



1, \ k x\y\ 

lo S FFTi ( 71 ) 



where flTDl comes from the fact that h(X\V,Y) is maximized by jointly Gaussian (V, X, Y), 
and (1711) comes from the monotonicity of | ■ | in positive semi-definite matrices. Now we 
introduce the following lemma. 

Lemma 2 

1 \K xlY \ 1 |K X | 1 |K X + S y | 

2 bg W = 2 l0g M| " 2 bg |F(D) + S y | (72) 

The proof of Lemma [2] is given in Appendix [El Lemma [2] and (ITT]) imply ([28]) . 
Next, we consider the mutual information leakage constraint in (j'24j) as follows 

J e >L(0,l)= min 7(V;X) -J(^;Y|m + /(X;Zim (73) 

(7^V^X-^Y,Z 

We note that the cost function of L(0, 1) can be rewritten as follows 

C(L) = I(V; X) - I(V; Y) + I(U; Y) + /(X; Z\U) (74) 
= J(F; X|Y) + [I{U- Y) + /(X; Z\U)} (75) 

where ( l74p comes from the Markov chain U — > V — > Y and ( I75p comes from the Markov 
chain V — > X — > Y. We note that the first term in (175|) is minimized by a jointly Gaussian 
(V, X) as we already showed in obtaining the lower bound for the rate given by ( 128]) above 
in (l66]) -( l7Tj) . On the other hand, the remaining term of (175]) in the bracket is maximized by 
a jointly Gaussian (U,X) as shown in [18J. Thus, a tension between these two terms arises 
if (U, V, X) is selected to be jointly Gaussian. In spite of this tension, we will still show that 
a jointly Gaussian (U,V,X) is the minimizer of L(0, 1). Instead of directly showing this, 
we first characterize the minimum mutual information leakage when (U, V, X) is restricted 
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to be jointly Gaussian, and show that this cannot be attained by any other distribution for 
(U, V, X). We note that any jointly Gaussian (U, V, X) can be written as 

V = AyX + Ny (76) 

U = AuX + Nu (77) 

where Ny,Nj/ are zero-mean Gaussian random vectors with covariance matrices Sy, S^, 
respectively. Moreover, Ny, are independent of X, Y, Z, but can be dependent on each 
other. Before characterizing the minimum mutual information leakage when (U, V, X) is 
restricted to be jointly Gaussian, we introduce the following lemma. 

Lemma 3 When D ^ Kx|y and V is Gaussian, we have the following facts. 

• Sy — D >- ; i.e., Sy — ~D is positive definite, and hence, non-singular. 

• We have the following equivalence: 

K X |yy^D K X |y^F(D) (78) 



The proof of Lemma [3] is given in Appendix [Fl Using Lemma EU the minimum mutual 
information leakage to the eavesdropper when (U, V, X) is restricted to be jointly Gaussian 
can be written as follows: 

L G = min I(V;X) - I(V;Y\U) + I(X;Z\U) (79) 

(7-s>y-5>X^Y,Z 
(C/,V,X) is jointly Gaussian 
K X |v^F(D) 

We note that the minimization in (J79il can be written as a minimization of the cost function 
in (1791 over all possible Au, Ay, S^, Sy matrices by expressing K. x \u and Kjqy in terms of 
Ajj, Ay, !][/, Sy. Instead of considering this tedious optimization problem, we consider the 
following one: 

TG ■ 1 i \ K x\ 1, iKxifz + Ey] 1 \K x \u + ^z\ , . 
L = mm -log-— r log ■— — ■ =— r + - log —-. (80) 

OdKxiv^K.^Kx 2 \Kx\v\ 2 & |K X |y + Sy| 2 & |E Z | V ' 

K X]V .^F(D) 

We note that due to the Markov chain U — > V — >■ X, we always have K^|y d ^-x\u- A 
proof of this fact is given in Appendix [Gj Besides this inequality, Kjqy and Kx\u might 
have further interdependencies which are not considered in the optimization problem in (180]) . 
Since neglecting these further interdependencies among JZx\u an d Kjqy enlarges the feasible 
set of the optimization problem in ( 179]) . we have, in general, 

L G > L G (81) 
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On the other hand, it can be shown that the value of L can be obtained by some jointly 
Gaussian (U, V, X) satisfying the Markov chain U — > V — > X, as stated in the following 
lemma. 

Lemma 4 

L G = L G (82) 

The proof of Lemma |4] is given in Appendix [H] 

Now we study the optimization problem LP in (|80|) in more detail. Let K^-.y and K^-,^ 
be the minimizers for the optimization problem L G . They need to satisfy the following KKT 
conditions. 

Lemma 5 If~K* x \ v and~K* x ^ u are the minimizers for the optimization problem IP , they need 
to satisfy 



K* X \v 


+ Sy)~ 1 +Mu 


+ M D = 


(K X \v 


r 1 


(83) 




(K^ + S^)- 1 


+ M X = 


(Fx\u 


+ Sy)- X +M V 


(84) 




Mu(K xlu - 


K*x\v) = 


<&c\u 


- K* xlv )Mu = 


(85) 




M D (F(D) - 


K*x\v) = 


(F(D) 


- K* xlv )M D = 


(86) 




M X (K X - 


*x\u) = 


(Kx- 


K* X]U )M X = 


(87) 



for some positive semi-definite matrices M(/,Mfl,Mj. 

The proof of Lemma [5] is given in Appendix [I] 

Next, we use channel enhancement [19]. In particular, we enhance the legitimate user's 
side information as follows. 

(K* xlu + Sy)- 1 = (K* xlu + Sy)- 1 + M, (88) 

This new covariance matrix Sy has some useful properties which are listed in the following 
lemma. 

Lemma 6 We have the following facts. 

• ^ Sy 

• Sy ^ Sy, Sy ^ S^ 

• (K* xlv + Sy)" 1 = (K* xl y + Sy)" 1 + M V 

. (K* xlu + £y)-\K x{v + Sy) = (K* x]u + Sy)- 1 ^^ + Sy) 

• {K* x]u + Sy)- X (K X + Sy) = (K x{u + S Z )- X (K X + S Z ) 
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. (K^ |y + S y )" 1 (F(D) + $y) = (K* X{V )^F(D) 

The proof of Lemma [6] is given in Appendix [J] Using this new covariance Sy, we define the 
enhanced side information at the legitimate user Y as follows 

Y = X + N y (89) 

where Ny is a zero-mean Gaussian random vector with covariance matrix Ey. Since we 
have Sy ^ Sy and Sy ^ as stated in the second statement of Lemma El without loss 
of generality, we can assume that the following Markov chain exists. 

X^Y^Y,Z (90) 



Assuming that the Markov chain in (1901) exists does not incur any loss of generality because 
the rate, mutual information leakage and distortion region TZ depends only on the condi- 
tional marginal distributions p(Y|X),p(Z|X) but not on the conditional joint distribution 
p(Y, Z|X). Now, we define the following optimization problem: 

L= min I(V;X) - I(V;Y\U) + I(X;Z\U) (91) 

(7->V-kX-+Y->Y,Z 

We note that we have I(V;Y\U) < I(V;Y\U) due to the Markov chain in (1901) . which leads 
to the following fact: 

L G = L G > L(0, 1) > L (92) 

Moreover, unlike the original optimization problem L(0, 1) in ( I73|) . we can find the minimizer 
of the new optimization problem L explicitly, as stated in the following lemma. 

Lemma 7 

1 \K X \ 1 \K x + t Y \ 1 |K X + S Z | 
L = o lo S rcVTvTT ~ o lo S 7^7^ S T + 7T lo § ( 93 ) 



2 °|F(D)| 2 °|F(D) + £y| 2 ° |S 



z\ 



We note that Lemma [7] implies that U = <fi and a Gaussian V leading to Kjqy = F(D) is the 
minimizer of the optimization problem L. The proof of Lemma [7] is given in Appendix iKl 
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Next, we show that indeed LP = LP = L which, in view of ( 192|) . will imply L(0, 1) 
L = IP = LP . To this end, using Lemma [7J we have 



1, |K X | 
2 l0g ^?Dl 



K 



- log , 
2 K 



x 



- log , 
2 K 



x\v\ 
K x \ 



- log , 
2 K 



x\v\ 



X\V\ 

lP 



1. |K X + £y| 1 
- log = l0£ 

2 S |F(D) + S y | 2 
|K x + £ y 




^x\v 



(94) 
(95) 

(96) 

(97) 
(98) 



where f l93|) comes from the last statement of Lemma O (I9"6"|) follows from the fifth statement 
of Lemma El and (197]) comes from the fourth statement of Lemma [6j In view of (|92|) . 
( 198|) implies that L(0, 1) = LP] completing the proof of Theorem H] as well as the proof of 
Theorem |5] due to the fact that Lf n = L(0, 1). 



5 General Case 

We now consider the general case where the side information are given by 

Y = HyX + Ny (99) 

Z = H Z X + N Z (100) 

where without loss of generality, we can assume that the covariance matrices of Gaussian vec- 
tors Ny and are given by identity matrices. We denote the singular value decomposition 
of Hy and Hz by Hy = QyAyRj and H z = Q^A^Rj, respectively Since any invertible 
transformation applied to the side information does not change the rate, information leak- 
age, and distortion region, the side information given by ({99]) - (11 00]) and the side information 
obtained by multiplying (199])- (11001) by Qy, Qj, respectively, yield the same rate, information 
leakage and distortion region. In other words, the side information given by (1991 -f llOOl) and 
the side information given by 

Y = AyRyX + Ny (101) 

Z = A z RjX + N z (102) 

yield the same rate, information leakage and distortion region, where the covariance matrices 
of Ny, Nz are given by identity matrices. Next, we claim that there is no loss of generality 
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to assume that the side information Y and Z have the same length as the source X. To 
this end, assume that the length of Y is smaller than the length of X. In this case, simply, 
we can concatenate Y with some zero vector to ensure that both Y and X have the same 
length. Next, assume that the length of Y is larger than the length of X. In this case, Ay 
will definitely have at least length(Y) — length(X) diagonal elements which are zero, and 
hence the corresponding entries in Y will come from only the noise. Since noise components 
are independent, dropping these elements of Y does not change the rate, information leakage 
and distortion region. Thus, without loss of generality, we can assume that length (Y) = 
length(X), and hence without loss of generality, we can assume that Ay is a square matrix. 
The same argument applies to the eavesdropper's side information, and hence, without loss 
of generality, we can also assume that A z is a square matrix. Next, we define the following 
side information 

Y a = (Ay + aI)RyX + Ny (103) 
Z a = (A z + aI)RjX + N z (104) 

where a > 0. We note that (Ay + al) and (Ay + al) are invertible matrices. Since 
multiplying the side information in (I1Q3I) -( TT00|) by some invertible matrices does not change 
the rate, information leakage and distortion region, the side information in f l 1 3 f) - ( TTQ4T) and 
the following side information 

Y Q = X + Ny Q (105) 
Z a = X + N Z , Q (106) 

have the same rate, information leakage and distortion region, where the covariance matrices 
of Ny a and Nz,a are given by 

Sy a = Ry(Ay + aI) _2 R.£ (107) 

S z , a = R z (A z + «I)- 2 Rj (108) 

respectively. For a given distortion constraint D, we denote the rate and information leakage 
region for the side information model given in (I99l-f ll00|) by 1Z (D), where the subscript o 
stands for the "original system", and for the side information model given in fll05p - fll06p by 
7Z a (D). We have the following relationship between 7Z (D) and lZ a (D). 

Lemma 8 

K CD) C limftjD) (109) 

The proof of Lemma [8] is given in Appendix [L] Next, using Theorem HJ we obtain an outer 
bound for the region lim^o 7Z a (D), where this outer bound also serves as an outer bound 
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for the region TZ (D) due to Lemma The corresponding result is stated in the following 
theorem. 



Theorem 6 J/D ^ Kjny, any (R,I e ) G 7£ (D) satisfies 



1 |K x , y | 1 |K X | 1 |H y K x H y + I| 
fl " 2 bg W = 2 bg jF^D)| ~ 2 l0g |H y F p (D)H y + I| ( 1 10) 



i e > mm o lo gTT7 r — o lQ g i i T ^ , , , ti + T^log |HyKx|C/H y + I| 



CMK^K^Kx 2 6 IKjcivI 2 |H y KxiyHv + 1| 2 

Kx|v^F (D) 



111) 



w/iere F (D) = (D" 1 - H y H y 



The proof of Theorem [6] is given in Appendix [Ml We prove Theorem [6] in two steps. In 
the first step, by using Theorem HI we obtain an outer bound for the region TZ a (D), and in 
the second step, we obtain the limit of this outer bound as a — > 0. As the outer bound in 
Theorem [6] basically comes from the outer bound in Theorem HI all our previous comments 
and remarks about Theorem 0] are also valid for the outer bound in Theorem El Similar to 
Theorem HI Theorem [6] also provides the minimum information leakage to the eavesdrop- 
per when the rate constraint on the transmitter is removed. Denoting the corresponding 
minimum information leakage by 7™ m (D), we have the following theorem. 

Theorem 7 //D ^ K-x\y, we have 

■ , x 1 lKv-1 1 |HyK X |r/Hj +11 1 -p 

/ p mm D > min -log , 1 Xl , --log 1 I + - log H y K X |t/H3 + 1 

e 1 ; " o<k x]v ±k x]u <k x 2 S \K X \ V \ 2 6 H y K X |yH y + I 2 Sl Y XW Y 1 



Kx|v^F (D) 



(112) 



where F (D) = (D^ 1 - H y H y ) 



-l 



As Theorem [TJ basically comes from Theorem [51 all our previous comments and remarks 
about Theorem are also valid for Theorem [71 



6 Conclusions 

In this paper, we study secure lossy source coding for vector Gaussian sources, where the 
transmitter sends information about the source in a way that the legitimate user can re- 
construct the source within a distortion level by using its side information. Meanwhile, the 
transmitter wants to keep the mutual information leakage to the eavesdropper to a minimum, 
where the eavesdropper also has a side information about the source. We obtain an outer 
bound for the achievable rate, mutual information leakage, and distortion region. Moreover, 
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we obtain the minimum mutual information leakage to the eavesdropper when the legitimate 
user needs to reconstruct the source within a certain distortion while there is no constraint 
on the transmission rate. 



Appendices 



A Proof of (ESS) 



We first define the following function 



R(D) = min I(X; V, Z) (113) 

V^X->Y,Z 



which is monotonically decreasing, continuous and convex in D. Next, we note that when an 
instantaneous encoding scheme is used, the minimum-mean-square-error estimator is given 
by 

Xi = E [X i \g 1 {X 1 ), . . . , g n (X n ), Y n ] (114) 
= E[X i \g i (X i ),YA (115) 

where (11151) comes from the independence of (X i: gi(Xi), Yi) across time. Consequently, when 
an instantaneous encoding scheme is used, the minimum-mean-square-error is given by 

a 2 Xi \ 9i(Xi)Yi = E[{X t - EiXMX,)^]) 2 ] (116) 

Assume that there exists an instantaneous encoding scheme that achieves the distortion level 
D: 

1 " 

lim -Y, a xM)Y l <D (117) 
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We now obtain a lower bound for the minimum information leakage for this instantaneous 
encoding scheme as follows 

lim -I(X n ;M,Z n ) = lim -I(X n ; g 1 {X 1 ), . . . , g n (X n ), Z n ) (118) 



1 - 

lim -V/^^i^),^) (119) 
i— >oo n z — * 

lim - V/pQ;!^) (120) 



n— >oo ?7, 

1=1 



1 n 

£„'™^E«W,|v,k,) (121) 

= *(^£°ii™) (123) 

> -R(-D) (124) 



where fll 19j) comes from the independence of (JQ, ^(JQ), Zj) across time, HI 20 j) follows by 
setting Vi = gi(Xi), f ll 21 p comes from the definition of R(D), (I122p is due to the convexity 
of R(D) in .D, (11231) follows from the fact that R{D) is continuous in D, and (I124p comes 
from (I117p and the fact that R{D) is monotonically decreasing in D. 



B Proof of Lemma [T] 

We first introduce two lemmas that will be used in the proof of Lemma HJ Throughout this 
appendix, we use notation A _LL B to denote U A and B are independent" to shorten the 
presentation. 

Lemma 9 Let Q, T, W be arbitrary random variables. If we have Q — )■ T — >■ T + W and 
T ALW. Then, we have (Q,T) il_ W 7 . 

Proof: Since a set of random variables is independent iff their joint characteristic function 
is the product of their individual characteristic functions, to prove Lemma [91 it is sufficient 
to show the following. 

E [ e -iQ+«T+«W] = E ^iQW] E J eS3 W] ^ V(si) (125) 
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We can show this as follows 



E [ e slQ+S2T+S3W ] =E[E [ e slQ+S2T+S3W/ |T]] (126) 

= E [e^- s ^ T E [ e '^+^ T + w )\T]] (127) 

= E [ e (s2 " S3)T E [e SlQ \T] E [e ss{T+w) \T]] (128) 

= E[e S2T E[e SlQ \T]E[e saW \T]] (129) 

= E [e S2T E [e SlQ \T] E [e s " w ] ] (130) 

= E [e S2T E [e SlQ \T]] E [e S3W ] (131) 

= E [ e SlQ+S2T ] E [e S3W ] (132) 



where fl 1 2 8 [) comes from the Markov chain Q — > T — > T + W and (I130p follows from the fact 
that T _LL W. Equation f |132[) implies the independence between (Q,T) and W; completing 
the proof of Lemma |9j ■ 

Lemma 10 Let Q,T,W be random variables satisfying (T,Q) _LL W and Q _LL T + W . 
Then, we have Q ALT . 

Proof: Similar to the proof of Lemma [91 here also we use the fact that a set of random 
variables is independent iff their joint characteristic function is the product of their individual 
characteristic functions. To this end, since (T, Q) _LL W, we have 

E [ e ^ w+S2T+s ^] = E [e s ' w ] E [ e S2T + s ^] , % Sl , s 2 , s 3 ) (133) 

If we set si = S2 in f ll33f) . we get 

E [ e S2H/+S2T+S3Q ] = E [e S2W ] E [ e S2T+s ^] , V(s 2 , s 3 ) (134) 

On the other hand, since Q _LL T + W, we have 

E [ e s * w + s * T + s &] = E [ e ^ w+T ^] E [e S3 °] (135) 
= E [e S2W ] E [e S2T ] E [e S3Q ] (136) 

where f ll36[) comes from the fact that T _LL W. In view of f !134j) and (I136p . we have 

E [ e S2T+S3Q ] = E [e S2T ] E [e S3Q ] (137) 

which implies that T _LL Q; completing the proof of Lemma dUJ ■ 

We now prove Lemma [TJ We note that we have I(V;Y\Z) = iff the Markov chain 
V — > Z — > Y holds. We prove by contradiction that when D < (tV, the Markov chain V — > 
Z — > Y is not possible. To this end, we note that the side information at the eavesdropper 
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can be written as 



X + N y + N z (138) 



or in other words, we have N z = N y +N z where N z is a Gaussian random variable independent 
of (X, N y ) with variance <j 2 z — a 2 > 0. Next, we note that the Markov chain V — > X — > Y — > Z 
implies (V, X) _LL (N y ,N z ) in view of Lemma EB Since Y,Z are jointly Gaussian, Y can be 
written as 

Y = aZ + (Y-aZ) (139) 

where a = E[YZ]/E[Z 2 ], and as a consequence of this a choice, we have Z AL Y — aZ. 
Hence, if we have the Markov chain 

V -> Z ->■ Y = aZ + (Y - aZ) (140) 

then, Lemma [9] implies that V _LL Y — aZ, where Y — aZ is 

Y - aZ = (1 - a)X + (1 - a)N y - N z (141) 

Since (V, X) _LL (N y , N z ), we have (V, X) AL (1 - a)JV w - N z , and also V JL (1 - a)X + (1 - 
a)A^ — X 2 due to the assumption that the Markov chain V — > Z — > Y holds. Hence, in view 
of Lemma [lOj we have V AL X. Moreover, since we have the Markov chain V — > X — > Y , 
V AL X implies that V AL (X,Y). Hence, if V AL (X,Y), we have a 2 x{vy = a 2 x{y . However, 
if D < & 2 i y , V AL X is not feasible, and this implies that the Markov chain V — > Z — > Y is 
not possible; completing the proof of Lemma [TJ 



C Proof of (EI) 



Here, we provide the proof of (I5ip . To this end, we consider a slightly more general case 
where the joint distribution of the source and side information is given by 



L 



y , z) = Yl P(xi, y h Zi) (142) 

i=l 

and the distortion constraint is imposed with a diagonal matrix D whose diagonal entries 
are denoted by D\, . . . , Dl. From Theorem |3l the minimum information leakage is given by 

J e min = min I{V;X)-I{V;Y\U)+I{X;Z\U) (143) 

C/->-V->-X->Y,Z 

a 2 , , 

XAVY L 
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We first introduce the following auxiliary random variables 



U^UY^Z^, i = l,...,L (144) 
Vi = VY i ~ x Xt. 1 , i = l,...,L (145) 



which satisfy the Markov chain 

Ui^Vt^Xi^ Y i} Z t (146) 



which follows from (11421) and the Markov chain U — > V — ?• X — >Y,Z. 
Next, we introduce the following two lemmas. 

Lemma 11 ([2j Lemma 7]) Let S n ,T n be length-n random vectors, and W be an arbitrary 
random variable. We have 

n n 

J^IiT^SilWS*- 1 ) = Y,HS l - 1 ;T i \WTl l +1 ) (147) 
i=i i=i 

Using Lemma [TT], the following lemma can be proved. 
Lemma 12 

n 

I(W;S n )-I(W;T n ) = ^I(W; S^S^T^) - I(W; T^S^T^) (148) 

i=l 
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Now, we proceed with (11431) as follows 



L min = min I(V;X)- I(V;Y\U) + I(X;Z\U) (149) 

a 2 , <Di, i=l,...,L 

min I(V; X) — I(V; Y) + I(U; Y) — I(U ; Z) + /(X; Z) (150) 

(7->V^X^Y,Z 
cr 2 r <Dj, i=l....,£ 



£ 

£ 



min X^Y* -1 , Xh.j) — I(V; Yj| Y 1 ' -1 , X. 

<£>i, i=l,...,£ 4=1 

■ \wL— ' ' ' 



+ J (U; Y\Y^\ Zf +1 ) - I{U- Z t \Y l -\ Z? +1 ) + /(X; Z) (151) 



i=l 

L 



m l n V7 ^/(V;X|Y-\X^-/(V^Y|^-\Xf +1 ) 



c/->v-»-x-»-y,z 

<Di, i=l,...,£ l ~ l 

a vv L— " ■ ' 



L 



mm 

'Xj|vy" 



+ ]T /(C/; F^Y- 1 , - I(U; Z t \Y^\ Z^ +l ) + /(X,; Z t ) (152) 

£ 

]T I(Y i ~ 1 , X, L +1 , V; X t ) - I(Y i ~ 1 , Xf +1 , V; Y t ) 



L 



+ 1{y^\Z^ U; Yj) - I(Y l -\ Z? +1 , U; Z { ) + I(X f , Z t ) (153) 



i=i 

L 

min V /(Vi; X) - J(^; Y) + Y) - /(C7<; ^) + I(X i; Z,) (154) 

f7-»V — >-X— >Y,Z 
CT 2 <£>. j=i £ * =1 

£ 

= „, v Xyz ^WiXJ-IiViMlUj + IiXtZilUi) (155) 

o- 2 <Di, i=l,...,£ i=l 

£ 

> min V/(\/ J ;X l )-/(V r i ;Y|^) + /(X;^|^) (156) 

f^i - >Vi— >X{— >Yi,Zi ' 
v 2 x . lv . Y .<Di, i=l,...,£ 1=1 

where (11501) comes from the Markov chain U — > V — ?■ X — > Y,Z, (I15ip follows from 
Lemma El ffT52|) and ffT53l are due to ([142]), ffT54l) follows from the definitions of U h Vi in 
(11441) and ()145p . respectively, (I155P comes from (11461) . and (I156P follows from 

a Xi\VYL > a Xi\VY^X^ +1 ( 157 ) 
= a X t \VY^X^ +1 ( 158 ) 

= *W, (159) 
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where (11571) follows from the fact that conditioning reduces MMSE (which will be shown in 
Appendix [G]), fl 1 5 8 [) comes from the following Markov chain 



XtMY* ^ ^ (160) 

which is a consequence of (11421) and the Markov chain U — > V — >■ X — > Y, Z, and (I159P 
is obtained by using the definition of V% given in ( 114511 . Hence, (11561) implies that when 
the joint distribution of the source and side information can be factorized as in (1142 j) . the 
minimum information leakage is given by 

L 

If n = min ^IiVi-X^-Iiy^Y^ + I^-Z^Ui) (161) 

<r 2 Xi]Vi y<Di, i=l,-,L ^ 

We now specialize (1 1 6 1 [) for the case given in Example [2J where L = 2 and we have the 
following Markov chains 

X a -> Y 1 -> Z x (162) 
X 2 ->• Z 2 ->• F 2 (163) 

Under these conditions, the minimum information leakage is given by 

J e min = min J(Vi;X 1 )-/(V 1 ;y 1 |l/ 1 ) + /(X 1 ;Z 1 |l7 1 ) 

Ui-*Vr-+X\-*Yi-+Zi 
v 2 XllVlYl <Di 

+ min 7(^ 2 ;X 2 )-/(^ 2 ;y 2 |[/ 2 )+/(X 2 ;Z 2 |[/ 2 ) (164) 

(72 -> Vi -+X 2 -»• Z 2 -> >2 

= min J(V 1 ;X 1 )-/(V x ;n)+/(^i;^i) 

Vi->Xi->-Yi->2i 

+ min I(y 2 ;X 2 )-/(y 2 ;Y 2 |^ 2 )+/(X 2 ;Z 2 |^ 2 ) (165) 

C/2 -> V2 -> X 2 -> Z 2 -> >2 

= min, /(^^-/(^YO+J^ZO 

+ min J(V 2 ;X 2 ) + J(X 2 ;Z 2 |\/ 2 ) (166) 



where (I165l) - (ll66p come from the following Markov chains 



t/i ->■ H ->■ X x ->■ Y 1 -> (167) 
C/ 2 -> V 2 -> X 2 -)• Z 2 -> y 2 (168) 
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respectively; completing the proof. 



D Proofs of (E2) and (j53j) 



We first prove (1521) . To this end, we note that when the joint distribution of the source and 
side information is given by 

L 

P(x,y,z) = Y[p(xi,yi,Zi) (169) 

i=l 

and the distortion constraint is imposed by a diagonal matrix D with diagonal entries 
Di, . . . , Dl, the minimum information leakage is given by 

L 

If n = min V/(y i ;X i )-/(^;F i |C/ i ) + /(X i ;Z i |[/ i ) (170) 

Ui-^Vi-i-Xi^Yi^i z — ' 



T 2 <n j=l,...,L i=1 



as shown in Appendix O (in particular, see (jlfiip ). When we set U = 4>, in other words, 
when we set U\ = 0, . . . , Ul = (ft, (I170p reduces to 

L 

4 mm " = v min ^/(^XO-Z^rO+JC^Zi) (171) 

°W 4 ^" i=l,.,L 1=1 

which is the desired result in (|52|) . 

Next, we prove ( !53l) by using ( 11701) . When we set U = V, in other words, when we set 
U 1 = V 1 ,...,U L = V L in (H7QD, we get 

L 

If a = min (172) 

which is the desired result in fl53l). 



E Proof of Lemma 2 



We note that since X, Y are jointly Gaussian, we have [201 P a g e 155] 



Kx|y — Kjjf — KxyK y 1 Kyx 

= K X -K X (K X + S y )- 1 K X 
= K x (K x + S y )- 1 S y 



(173) 
(174) 
(175) 
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where (I174p comes from the fact that Y = X + Ny. Next, we have the following chain of 
equalities 



\K x (K x + X Y )- l \ [K^Kx + Syj^gyj 

|F(D)(F(D) + Sy)" 1 ! |F(D)(F(D) + Sy^Sy] 



IK 



X\Y\ 



|(Ey(Ey - D)- 1 Sy - Sy)S y X (Sy - D) 

|Kxiy| 



IDI 



(176) 
(177) 
(178) 



where ffTTTj) follows from the definition of F(D), i.e., F(D) = S y (S y - D)- x Ey - Sy. 
Equation (11781) implies (172"]) ; completing the proof of Lemma [2J 



F Proof of Lemma [3 



We first prove the first statement of the lemma. To this end, using (11751) . we have 

K X[Y = K X (K X + Sy)- 1 Sy (179) 

= Sy-Sy(K X + Sy)^Sy (180) 



Hence, using (I180p . the constraint D ^ Kx|y can be expressed as 

D r< Sy - Sy(K X + Sy) _1 Sy 



;i8i) 



which is 



Ey(K X + Sy^Sy ^ Sy D 



182) 



where Sy(K^ + Sy) x Sy >- implying Sy — D >- 0. Hence, Sy — D is non-singular, and 
(Sy - D)- 1 exists. 

Next, we prove the second statement of the lemma. To this end, we note that since 
(V, X, Y) are jointly Gaussian, Y = X + Ny, and V is independent of Ny, Kx|yy is given 
by [201 page 155] 



K 



X\VY 



K 



x 



K xv K x } M- 1 [ K xv K x ] T 



(183) 



where M is given by 



M 



K vx 

Ky 



;i84) 
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Using block matrix inversion lemma J2TJ page 45], M 1 can be obtained as 



rv/T 1 = 

where A M is given by 



Ky-KwKr/K, 



(185) 



= K.v-K. vv K r , K rA H £ 



Y 



(186) 
(187) 
(188) 



where the last equality follows from the fact that Kx|y = — K^yK^/Kyx- Using (11851) 
and (jlggjl . we get 



[ K xy K x ] M- 1 = [ XyA^KxyKy 1 I - EyA^ 1 ] 



using this in conjunction with (11881) . we obtain 



K xv K x } M- 1 [ K xv K x ] T = K x - Ey + EyA^Ey 



(189) 



(190) 



Using ([1901) in (jlggj) . we have 



Kx|yy — Ey — EyA M Ey 



Sy — Ey(Kx|V + Ey) *£ 



Y 



where (11921) follows from (11881) . Thus, using (11921) . the constraint K 
expressed as follows 

Ey - £y(K X |y + Ey)" 1 Ey ^ D 



•XI VY 



(191) 
(192) 

^ D can be 



(193) 



from which, since Ey — D y 0, the following order can be obtained 



Kxiy =< S y (Ey - D)- 1 E y - Ey = F(D) 



which completes the proof of Lemma [31 



(194) 



G Conditioning Reduces MMSE 

Here, we prove that conditioning reduces MMSE. To this end, we introduce the following 
lemma. 
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Lemma 13 Let U and V be any two n- dimensional random vectors and g : W 1 — >■ M n . 



E [g(V)g T (V)\U = u] h E [g(V)\V = u]E [g J (V)|U = u] 



(195) 



Proof: The proof of this lemma comes from the following fact 



-< E 



(g(V) - E [g(V)\U = u]) (g(V) - £ b(V)|U = u]) T |U = u 



E [g(V)g T (V)\V — u] — E [g(V)\U = u]E [g T (V)\V = u] 



(196) 
(197) 



We now prove the fact that conditioning reduces MMSE. 
Lemma 14 IfU-t V — >• X, #ien K X |y d K X |c/- 



Proof: We have 



K X | V = £ [XX T ] - £ [E [X|V] £ [X T |V]] 

= E [XX T ] [X|V] E [X T |V] |U]] 

r< E [XX T ] - £7 [£7 [E [X|V] |U] £ [E [X T |V] |U]] 

= £7 [XX T ] — E[E [X|U] £ [X T |U]] 

where ( j200[) comes from Lemma [13] and ( 120 1}) comes from the following fact 



(198) 
(199) 
(200) 
(201) 



E [E [X|V] |U] = E [X|U] 



(202) 



which is a consequence of the Markov chain U — > V — > X. 



H Proof of Lemma 4 



We now prove Lemma HI Since any jointly Gaussian (U, V, X) triple satisfying the Markov 
chain U — > V — > X also satisfies Kx|y d Kx|t/ due to Lemma [HJ the feasible set of L G 
already contains all jointly Gaussian (U, V) pairs satisfying the Markov chain U — > V — > X. 
Hence, we have L G > L G . Next, we show that L G > LP to complete the proof of Lemma HI 
To do so, we need to show that for any jointly Gaussian (U, V, X) with conditional covariance 
matrices Kx\u an d K x \v satisfying H K x \v d> ^x\u d Kx and K. X \ V d F(D), there exists 
another jointly Gaussian (£/ G , V G ) pair such that this pair has the following properties 

• ^-x\u a = K x \u 
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• U G ->■ V G ->■ X 
To this end, we note that (£7 G , V G ) can be represented as 

V G = AyX + Ny (203) 

f/ G = Af/X + Nj; (204) 

where (Nj/, Ny) and X are independent, Njy, Ny are zero-mean Gaussian random vectors 
with identity covariance matrices. The cross covariance of N[/ and Ny is given by Xj/y — 
E [NyNy] , which needs to be selected accordingly to ensure that U G — >■ V G — >■ X. 
The conditional covariance K X |yG is given by [201 P a g e 155] 

K x , yG =K X - KxvaK^Kyax (205) 

Since we are seeking a V G such that K X |yG = K. X \ V , we set K X |yG = Kx|y in (I205j) yielding 

K x ,y = K x - K^K^K^x (206) 

= K X - K x A y (AyK x A y + IJ^AyKx (207) 

which is equivalent to 

K X \K X - K X |y)K^ = A y (AyKxAy + 1)"%, (208) 

Next, we note the Woodbury matrix identity [22J. 
Lemma 15 ([22], page 17]) 

(A + CBC T ) _1 = A" 1 - A X C (B^ 1 + C T A _1 C) _1 C T A" 1 (209) 

Using Woodbury matrix identity, we get 

(AyK^Ay + I)" 1 = I - Ay(K- x + A^Ay)- x A^ (210) 

using which in (12081) . we get 

K X \K X - K X |y)K^ = Ay [I — Ay(K x x + Ay Ay) -1 Ay] Ay (211) 

= Ay Ay - AyAy(K^ + AyAy) _1 AyAy (212) 

= Ay Ay - AyAy(K x 1 + AyAy)^ (K^ 1 + Ay Ay - K x x ) (213) 

= AyAy(K x x + AyAy)- 1 ^ 1 (214) 

= (K^ + AyAy - K x x ) (K x x + AyAy)- 1 ^ 1 (215) 

= K^ 1 - K-^K- 1 + AyAy)- 1 ^ 1 (216) 
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which implies 

K xlv = (K^ + A^Ay)" 1 (217) 

which, in turn, implies 

A T V A V = K x ] v - K x x (218) 

Hence, if we select Ay as satisfying (12181) . we get K x \v G — ^x\v- Similarly, if we select A v 
to satisfy 

A T V A V = K-| a - (219) 

then, we also have K. x \u G = ^x\u- 

Next, we will explicitly construct Ay and Au matrices to satisfy (12 181) and (12 19j) . re- 
spectively To this end, we introduce the following lemma, which will be used subsequently. 



Lemma 16 ( |23| ) Let A,B be two real symmetric positive semi- definite matrices. Then, 
there exists a non-singular matrix W such that 

A = W T A A W (220) 
B = W T A B W (221) 

(222) 

where A a and Ab are diagonal matrices. 

Lemma [TBI states that two real symmetric positive semi-definite matrices can be diagonalized 
simultaneously. Using this fact in (l218j) -f |2T9|) . we get 

K x ] v - K x x = W T AyW (223) 
K-}^ - K x x = W T AlW (224) 

for some non-singular matrix W, and diagonal matrices Au, Ay. Since K~x\v ^ ^-x\u, we 
have K~} y h K"}^, which, in view of (12231) -( 12241) imply 

W T (Ay - A v ) W y (225) 
Since W is non-singular, (I225j) implies that 

Ay h A v (226) 
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Finally, we choose 



A y = A y W (227) 
Au = A V W (228) 

which, in view of fl2T8l)-fl2T9l) and fl223D-fl224D. imply K x \ v g = K x \v and K X \ua = K x]u . 

Next, we show that a proper selection the cross- covariance matrix "Suv would yield the 
desired Markov chain U G — > V G — > X. To this end, we introduce the following matrix 



*-uv 



A V A V ( 229 ) 



where the diagonal matrix Ay is defined as follows: 



v 

i 



if A VAi ^ 



A Vtt ={ v > nr " (230) 

I 0, otherwise 

Since Au ^ Ay, we have Aj/AyAy = Au- Hence, we have 

A UV A V = A v (231) 

We also note the following 

A uv A uv = A v (Ay) 2 Au±I (232) 

since Au d Ay. 

Now, we are ready to show that U G and V G satisfy the Markov chain U G — > V G — > X 
by specifying S^y. We set Nu as follows 

Nu = Af/yNy + N (233) 

where N is a zero- mean Gaussian random vector with covariance matrix I — A^/yA^y, and 
is independent of Ny. In view of (I233p . we have 

U G = Af/X + Nu (234) 

= Af/yAyX + A^yNy + N (235) 

= A uv V G + N (236) 
which implies that (U G , V G ) satisfy the Markov chain U G — > V G — > X; completing the proof. 
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I Proof of Lemma 



5 



The Lagrangian for the optimization problem LP is given as follows 

£ ( L = o log 1 ~ n log \v iv I + o log tr(M K X |y) 

- ti(M v (K X \u - Kx|y)) - tr(M x (K x - K X \u)) ~ tr(M D (F(D) - K x[v )) 

(237) 

where the positive semi-definite matrices M , My, M^, Mj are the Lagrange multipliers for 
the following constraints 

K x , y h (238) 
- K x{v y (239) 
F(D) - K xlv h (240) 
Ki - K x{u h (241) 

respectively. Let and be the minimizers of the optimization problem LP . Using 

(12371) . the KKT conditions can be found as follows. 

Vk x|v £(I G ) | Kjc|v =k V = (242) 

V Kxlu C(L G ) \k xiu =k* x[u = (243) 

tr(M K x|y ) = (244) 

tr(M C7 (K x|[7 -K x|y )) = (245) 

tr(M D (F(D)-K x|y )) = (246) 

tr(M*(K x -K x|[7 )) = (247) 

We first note that we have K X |y >- 0, otherwise IP — > oo. Hence, using the fact that if 
A y 0, B y 0, tr(AB) > 0, and (12321) . we get M = 0. Next, using the fact that M = in 
(12421) . we get the KKT condition given in (1831) . Equation (I243P implies (1841) . Finally, using 
the fact that A y 0, B h 0, tr(AB) = tr(BA) > in ([245]) - (12471). we can get the KKT 
conditions given in (J85l)-(l871). respectively. 
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J Proof of Lemma 



6 



We start with the second statement of the lemma. To this end, we note that (15111 and ( 1881) 
imply the following. 

(K* xlu + Sy)- 1 = (K* xlu + Sy)- 1 + M v (248) 
= (K X{U + 'E Z )- 1 + M X (249) 

Next, using the fact that if A y 0, B >~ and A y B, we have A^ 1 ^ B -1 in conjunction 
with the fact that Mu y 0, Mx b 0, we can obtain the second statement of the lemma from 

fl2H-(T549D. 

Next, we consider the third statement of the lemma as follows 



= K X[V + [(K* xlu + Sy)- 1 + M,;] 1 - K* xlu (250) 

= K xlv + [I + (K* xlu + ^y)Mu] - 1 (K x]u + Sy ) - K* x]u (251) 

= K V + [I + (K xlu -K* x]v + K* xlv + Ey)M C7 ] 1 (K* xlu + Sy) - K x]u (252) 

= K V + [I + (K V + Sy)M c/ ] - 1 (K^ + Sy ) - K^ |[7 (253) 

= K* xlv + [(K xlv + Sy)" 1 + Mu] 1 (K* x]v + V Y r l (K xlu + Sy) - JC xp (254) 
= K xlv + [QC X]V + Sy)- 1 + M v ] - 1 (K x]v + V Y )- l (K xlu -JC X]V + K* x{v + Sy) 

- K'x\u (255) 
= K* xlv + [(K* xlv + Sy)- 1 + M^] 1 (K* x]v + Sy)- 1 ^ - K x]v ) 

+ [(K x]v + Sy)" 1 + M v ] - 1 - K* x{u (256) 
= K x]v + [(K* xlv + Sy)- 1 + M,;] 1 [(K* xlv + Sy)- 1 + Mu] (K x]u - K* x]v ) 

+ [(K* xlv + Sy)- 1 + Mu] 1 - K* x{u (257) 

= K* xlv + (K x]u - K* x]v ) + [(K xlv + Sy)- 1 + Mu] 1 - K* xlu (258) 

= [(K^ + Sy^ + M^f 1 (259) 



where (12501) comes from fl248|) . (T253|) and (1257]) follow from (jg5]l . 
Now, we consider the fourth statement of the lemma as follows 

(K* xlu + t Y )-\K xlv + Sy) = I + (K^ + Sy)- 1 ^ - K* x]u ) (260) 

= I + [(K* x]u + Sy)- 1 + Mv] (K* x]v - K* x]u ) (261) 

= I + (K* x]u + X Y )-\K X]V - K* x]u ) (262) 

= (KW + Sy)- 1 (KW + Sy) (263) 



where (12611 follows from (12181) . and (12621) comes from (JB5). 
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Next, we consider the fifth statement of the lemma as follows 

(K* x]u + t Y )-\K x + XV) = I + (K x]u + Sy)- 1 ^ - K x]u ) (264) 

= I + [(K x]u + Hz)- 1 + M X ] (K x - K xlu ) (265) 

= I + (K* xlu + S,)- 1 ^ - K xlu ) (266) 

= (K^ |[7 + S Z )- 1 (K X + S Z ) (267) 

where (12651) comes from fl249|) . and fl266l) is due to (1571) . 

Now, we prove the last statement of the lemma. To this end, we note that the third 
statement of this lemma and (1831) imply the following 

(K* xlv + Sy)- 1 + M D = (K*^)- 1 (268) 

which will be used in the sequel. Now, the last statement of this lemma follows from 

(K* xlv + Sy)' 1 (F(D) + Sy) = I + (JC X]V + Sy)- X (F(D) - K^ |y ) (269) 

= I + [(K^)- 1 - M D ] (F(D) - K* x{v ) (270) 

= I + (KVr 1 (F(D)-%) (271) 
= (K^ |y )- 1 F(D) (272) 

where (12701 comes from (12681) . and fl27H is due to (|8"B1). 

Finally, we note that (12681) also implies the first statement of the lemma; completing the 
proof. 

K Proof of Lemma [71 
K.l Background 

We need some properties of the Fisher information and the differential entropy, which are 
provided next. 

Definition 1 ([24, Definition 3]) Let (U, X) be an arbitrarily correlated length-n random 
vector pair with well-defined densities. The conditional Fisher information matrix o/X given 
U is defined as 

J(X|U) = E [p(X|U)p(X|U) T ] (273) 
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where the expectation is over the joint density /(u, x), and the conditional score function 
p(x|u) is 

p(x|u) = Vlog/(x|u) = 

We first present the conditional form of the Cramer-Rao inequality, which is proved 
in [23]. 

Lemma 17 (| I24L Lemma 13]) Let U, X be arbitrarily correlated random vectors with well- 
defined densities. Let the conditional covariance matrix of X be Cov(X|U) y ; then we 
have 

J(X|U) y Cov(XlU)- 1 (275) 

which is satisfied with equality if (U, X) is jointly Gaussian with conditional covariance 
matrix Cov(X|U). 

The following lemma will be used in the upcoming proof. The unconditional version of 
this lemma, i.e., the case T = <f), is proved in [23} Lemma 6]. 

Lemma 18 ( \24[ Lemma 6]) Let T, U,Vi,V2 be random vectors such that (T, U) and 
(Vi, V2) are independent. Moreover, let Vi, V2 be Gaussian random vectors with covariance 
matrices Si, S2 such that -< Si -< S2. Then, we have 

J- X (U + V 2 |T) - S 2 y r\\J + Vi|T) - S x (276) 

The following lemma will also be used in the upcoming proof. 

Lemma 19 (| I24L Lemma 8]) Let Ki,K 2 be positive semi-definite matrices satisfying -< 
Ki ^ K 2; and f(K) be a matrix-valued function such that f(K) y for Ki -< K -< K 2 . 
Moreover, f(K) assumed to be gradient of a scalar field. Then, we have 

/ f (K)dK > (277) 

The following generalization of the de Bruijn identity [25, 26] is due to [27], where the un- 
conditional form of this identity, i.e., U = <f), is proved. Its generalization to this conditional 
form for an arbitrary U is rather straightforward, and is given in [2U Lemma 16]. 

Lemma 20 ([24, Lemma 16]) Let (U, X) be an arbitrarily correlated random vector pair 
with finite second order moments, and also be independent of the random vector N which is 



gjog/Qxju) 



dlog/(x|u) 

dx n 



(274) 
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zero-mean Gaussian with covariance matrix S^r >~ 0. Then, we have 

V Sjv /i(X + N|U) = ^J(X + N|U) (278) 

The following lemma provides a connection between the conditional covariance matrix 
and the Fisher information matrices of a random vector. 

Lemma 21 Let (V, X) be two arbitrary random vectors with finite second moments, and 
N be a zero-mean Gaussian random vector with covariance matrix S^. Let Y = X + N. 
Assume (V, X) and N are independent. We have 

(X + N|y)Ejv (279) 

Lemma [21] is proved in [27] for V = <fi. Its generalization to the current conditional form can 
be obtained by using the conditional Fisher information and Lemma [20] 



K.2 Proof 

We first consider the cost function of the optimization problem L 

C{L) = I(V; X) - I(V; Y\U) + /(X; Z\U) (280) 

= I{V- X) - I(V] Y) + I{U- Y) + /(X; Z) - /([/; Z) (281) 

= I(V; X) - I(V; Y) + I(U; Y, Z) + /(X; Z) - I(U; Z) (282) 

= I(V; X) - I(V; Y) + I(U; Y|Z) + /(X; Z) (283) 

>I(V;X)-I(V;Y) + I(K;Z) (284) 

where (1281 II - fl 2 82 p come from the following Markov chain 

£/^V^X^Y^Y,Z (285) 

and (12841) comes from the non-negativity of the mutual information. On the other hand, 
(12841) can be obtained from f[9T|) by choosing U — <p, i.e., we have 

L< min J(V;X)-J(V;Y) + J(X;Z) (286) 

V->X->Y->Y,Z 

K x]VY ±r> 
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Hence, (I284p and (I286P imply the following 



L 



mm 

y-»X->Y->Y,Z 



I(V;X)-I(V;Y) + I(X;Z 



(287) 



mm 

V->X->Y-s.Y,Z 



I(V;X\Y) +/(X;Z 



(288) 



where ( 12881) comes from the Markov chain V — > X — > Y. We note that the optimization 
problem in ( I288p is similar to the one we already studied in ( 1671) -( I7TI) . Indeed, if the constraint 
K;qyy ^ D in (I288p was K. x ^ V y d D, both optimization problems would be identical, and 
using the analysis in (1671) - (17T1) . we could conclude that (I288P is minimized by a Gaussian V 
satisfying K. x ^ V y ^ D. However, the difference between these two constraints necessitates a 
new proof, and indeed, showing the optimality of Gaussian V for the optimization problem in 
( 12881) is not as straightforward as showing the optimality of Gaussian V for the optimization 
problem in (J67J. 

We find the minimizer for the optimization problem L in two steps. In the first step, for 
a given feasible V, we explicitly construct a feasible Gaussian V which provides the same 
value for the cost function of L as the original V does. Thus, this first step implies that 
restricting V to be Gaussian does not change the optimum value of the optimization problem 
L. Consequently, in the second step of the proof, we minimize L over all feasible Gaussian 
V. To this end, we note that the cost function of the optimization problem L can be written 
as 



for some constant c, which is independent of V. From now on, we focus on the difference of 
the two differential entropy terms in ( I289p . Next, we note that using Lemma 1201 we have 



where N is zero-mean Gaussian random vector with covariance matrix S^r satisfying ^ S^. 
Next, we find upper and lower bounds for (I290p . We note that Lemma[T8limplies the following 
upper bound for J(X + N|V) 



C(L) = h(Y\V) - h(X\V) + c 



(289) 




(290) 



J(X + N\V) r< [J _1 (X|V) + Sjv] 



-1 



(291) 



Using (12911) in (I290p in conjunction with Lemma [T9"l we get 



h(Y\V) - h(X\V) < -log 



J-^XIVQ + S 
\J-i(X\V)\ 



(292) 
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We note that due to Lemma [T7J we have J(X|V) >z K x j y >- 0, i.e., (1292 j) is well-defined. 
Similarly, using Lemma [T8~| we have 

j _1 (x + n y \v) - Sy y j -1 (x + N|y) - s^, s w ^E y (293) 

which implies 

J(X + N|F) h [j-^X + Nyly) -Ey + Ejv] 1 (294) 
Using (12941) in ( 1290p in conjunction with Lemma [T9l we get 

MV|V)-MX|,)^io 8 | ^ x '^^ y| w, 

Now, we rewrite the bounds in (1292 j) and (I295p . To this end, we define the following function 

^'W' °- f - 1 (296) 

where the matrix K(i) is given as follows 

K(t) = tJ _1 (X|V) + (1 - t) [j _1 ( x + N Y \V) - E y l (297) 

Hence, using f(t) in (I296p . the bounds in (12921) and (12951) can be rewritten as follows: 

f(0)<h(Y\V)-h(X\V)<f(l) (298) 

Since /(£) is continuous in t, there exists t* such that 

f(t*) = h(Y\V)-h(X\V) (299) 
1, |K(r) + I 

= — log 

2 8 |K(t*)| 

where K(t*) is bounded as follows 



lo § J^m (300) 



J _1 (X|^) r< K(t*) d J _1 (X + Ny|V) - Sy (301) 
r< J _1 (X + Ny|F) -Ey (302) 

where we used the fact that < t* < 1 and Lemma Thus, (I300p implies that if we pick 
a Gaussian V satisfying K X |y = K(i*), it provides the same value for the cost function of L 
as the original V does. 

Next, we check whether this Gaussian V is feasible, i.e., whether it satisfies ~K-x\vy 

-< D. 
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To this end, using Lemma EU we get 



K xl y Y = Sy - SyJ(Y|V)Sy (303) 

Since V is Gaussian, Lemma [T7] implies that 

J(Y|V) = (304) 
= (K x ,y + Sy)- 1 (305) 

where f l305[) follows from the fact that (V", X) and Ny are independent. Moreover, due to 
(13020 ; we have K X |y ^ J _1 (Y|V) -Sy, which together with f l305|) imply the following 



J{Y\V) y J(Y\V) (306) 



Using (13061) in fl303|) . we get 



K XI vy ^Sy- S y J(Y|V)Sy (307) 
= K X | Vy (308) 
r< D (309) 

where ( 13081) follows from Lemma I2T1 and ( 1309[) is due to the assumption that V is feasible, 
i.e., Kx|yy ^ D. Equation ( 1309|) implies that the constructed Gaussian random vector is 
feasible, i.e., for each feasible V, there exists a feasible Gaussian which provides the same 
value for the cost function of L; completing the first step of the proof. 

Hence, in view of this first step of the proof, we can restrict V to be Gaussian which 
leads to the following form for L: 

L= min I(V;X) - I(V;Y) + /(X; Z) (310) 

f^X->Y->Y,Z 

V is Gaussian 

min I(V;X) - I(V;Y) + /(X; Z) (311) 

V is Gaussian 
K X |v^F(D) 

1 \K X \ 1 |Kx+jr| 1, |K X + S Z | 
= mm -log-; r log ~ — log — — (312) 

K X|V ^F(D) 2 & |K X |y| 2 & |K X |y + Sy| 2 & |E Z | 

1 \K X \ 1 |K X + Sy| 1 |K X + S Z | 
= n lo S ir/^r o 1 °g l ^ . ^ , + o lQ g — pf; i — ( 313 ) 



2 ° F(D) 2 °|F(D) + E 
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where ( 13 11 P follows from Lemma El and (13131) comes from the fact that 



IK 



x\v\ 



(314) 



is monotonically decreasing in the positive semi-definite matrices K.x\v] completing the proof 
of Lemma H 



L Proof of Lemma [8 

We note that due to Theorem [31 we already have single-letter descriptions for the regions 
7Z (D) and lZ a (D). Thus, to prove Lemma El it suffices to show that for any given feasible 
(U, V), these two regions satisfy the relationship given in Lemma We first note the 
following Markov chains 

f/^V^X^Y Q ^Y (315) 
U^V^K^Z a ^Z (316) 

Next, we show that any feasible (U, V) for the region TZ (D) is also feasible for the region 
lim Q ^o 7£a(D)- To this end, we note that 

D h K xlVY (317) 
h ^x\VY9 a (318) 
= Kx\v% (319) 

where f 1 3 1 8 [) is due to the fact that conditioning reduces MMSE and ( 131 Qf> follows from the 
Markov chain in (131 5p . Moreover, it can be shown that \im a ^.oK. x , v f exists and is equal 
to K x \vy- Hence, this observation and (13 19[) imply that (U, V) is also feasible for the region 
lim Q ^ TZ a (D). 

Next, we show that for a given (U,V), any rate inside the region TZ (D) is also inside 
lim^o TZ a (D). To this end, for a given (U, V), we denote the minimum achievable rates in 
7Z (D) and lZ a (D) by R Q and R a , respectively Due to Theorem [3J, we have 

R a -R a = [I(V; X) - I(V; Y)] - [I(V; X) - I(V; Y a )} (320) 

= I(V;Y a )-I(V;Y) (321) 

= /(y;Y a |Y) (322) 

> (323) 

where (I322p comes from the Markov chain in (I315p . Equation (I322j) implies that any achiev- 
able rate within the region TZ (D) is also included in the region lim^o 1Z a (D). 
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Finally, we show that for a given (U,V), any achievable information leakage inside the 
region TZ (D) is also inside lim^o 1Z a (D). To this end, for a given (U,V), we denote the 
minimum information leakage in 1Z (D) and TZ a (D) by J ej0 and I e>a , respectively. Due to 
Theorem [3J we have 

h,o ~ h,a = [I(V; X) - I(V; Y\U) + /(X; Z\U)] 

- [l(V;X)-I(V;Y a \U) + I(X;Z a \U)] (324) 

= [l(V;Y a \U) - I(V;Y\U)] + [/(X; Z\U) - J(X; Z a \U)] (325) 

= I(V; Y a \U, Y) + [/(X; Z\U) - /(X; Z a \U)] (326) 

> I(K;Z\U) - I(X;Z a \U) (327) 

> /(X; Z) - /(X; Z a ) (328) 

1, |W ^„ T T| 1 \K x + R z (A z + aI)- 2 R T z \ 
= - log \H Z K X H Z + 1| - - log |Rz(Az + aI) - 2R T, ( 329 ) 

- 1 log |H,K*HJ + 1| 4 log + R ^ (A - + ^J^* + ^51 



2 1 2 6 |R z (A z + aI)- 1 QTQ z (A z + «I)- 1 R5| 



(330) 

i log |H z K x Hj + 1| - X - log |Q z (Az + «I)R^K X R Z (A Z + aI)Qj + I| 

(331) 



where (I326p comes from the Markov chain in (13151) and (I328p follows from the Markov chain 
in (13161) . Equation (13311) implies that 



h,a > ~ log 




+ I| 


— lim - 

a^O 2 


log |Qz(A z + aI)R T z K x R z (A z 


+ aI)Q T z + I\ 

(332) 


= l hg 


|H z KxH x 


+ I| 


-5 log 


Q Z \ Z R T Z K X R Z \ Z Q T Z + l\ 


(333) 


= 2 log 




+ I| 


-~log 


H z K x Hj + I| 


(334) 


= 










(335) 



where (I333p comes from the continuity of the determinant in positive semi-definite matrices. 
Equation (133 5p implies that any achievable information leakage in the region TZ (D) is also 
inside the region lim^o lZ a (D); completing the proof of Lemma [H 



M Proof of Theorem [6 

We start the proof of Theorem [6]by first expressing Theorem H]for the side information model 
given by (I105p - (11061) . In other words, we first provide an outer bound for the region lZ a (D) 
by using TheoremHl To this end, to be able to use Theorem!!], we need D H K^,^. However, 



44 



since we originally have D ^ Kjqy and ~K. X \f — K-x\y, where the latter one follows from 
the Markov chain X — > Y a — > Y and the fact that conditioning reduces MMSE, K X |y — D 
might be indefinite. However, the only place we use the condition D ^ K~x\y is to be able to 
show the equivalence between K x \vy d D and Kx|y ^ F(D) for Gaussian V in Lemma El 
In particular, we only need the fact that Ey — D is non-singular to show this equivalence, 
and which is implied by D ^ K^iy- However, still there might be distortion matrices D 
for which although we have non-singular Sy — D, the condition D ^ Kx|y is not satisfied. 
Hence, if we can find an a* such that 



S Y)Q - D^O, 0<a<a* (336) 

we can still use Theorem H]to obtain an outer bound for the region TZ Q (D). Now, we establish 
the existence of such an a*. Using the assumption D ^ Kx|y, we have 

D ■< K x \y = (Kj 1 + HyHy) 1 (337) 

where the equality follows from (12171) . Equation (13371) implies that 

-< D 1 - H y H y (338) 
= D 1 - RyA y Rj (339) 

where we use the singular value decomposition of Hy. Thus, since D" 1 — RyAyRy is strictly 
positive definite, there exists < (3 such that 

D 1 - RyA y Ry y (3 2 I (340) 
= /3 2 R y Rj (341) 

which implies 

D 1 y Ry(A y + /3 2 )Rj (342) 

which, in turn, implies the existence of an a* such that 

D 1 y Ry(A y + a) 2 Rj, < a < a* (343) 

Hence, using the definition of Sy a in (13431) . we get 

D 1 y Sy^, < a < a* (344) 

which is equivalent to the desired condition in (13361) which is needed to use Theorem |4] to 
obtain an outer bound for the region 7Z a (D). Hence, assuming that < a < a*, an outer 



45 



bound for the region TZ a (D) can be written as the union of rate and information leakage 
(R, I e ) pairs satisfying 



R > - log 

~ 2 e 



IK 



X\Y Q 



IDI 



1, \ K x\ 



F„(D] 



J e > mm - log — — 

O^Kxiv^Kx^^Kx 2 |K 
K X |v^F a (D) 



X|V| 



1. |K X + £y Q | 

— log ■ 

2 & |F Q (D) + £y Q | 

1 , |Kx|(7 + Sy Q | 

- - log i ! r 

2 |Kx|V + Sy >a | 



2 log 



IK 



x\u 



J Z,a\ 



(345) 
(346) 



where F a (D) = Sy )Q ,(Sy jQ , — D) _1 Sy )Q , — £y a . We now find the limiting region that comes 
from the one described by (I345I) - (I346I) as a — > 0. To this end, we introduce the following 
lemma that will be used subsequently. 

Lemma 22 



lim K^.y 
lim F a (D) 

a— >0 



Kx|y 

(D -1 — HyHy) -1 



(347) 
(348) 



The proof of Lemma [22] is given in Appendix [N] 



We first consider the rate bound in 



as follows 



lim - log 

a-*0 2 



IK 



X\Yj 



|D| 



1 IK 
— log 

2 D 



X\Y\ 



(349) 
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which follows from the continuity of the determinant in positive semi-definite matrices and 
(I347p . Similarly, for the second expression in the rate bound in (I345p . we have 



lim - log 

a->0 2 



IK 



-V I 



FJB) 



log • 

2 & 



Y,a\ 



FJT» 



J Y,a\ 



log 



log 



- log 

2 & 



log 



log 



\ log 



D 



-l 



D 



-l 



D 



D 



K 



x 



HyHy) 1 



K 



X 



HfH y )-i 



K 



x 



HfH y )-i 



K 



x 



HfHy) 1 



K 



X 



HfHy)" 1 



D 



HfHy)- 1 



lim - log 

a-s-0 2 



lim - loe 

a^o 2 



lim - log 

a^O 2 



— lim - lo 



|Kx + EyJ 



(350) 



(351) 



|F a (D) + £y, a | 

|Kx + Ry(Ay + aI)^ 2 Ry| 
|F Q (D) + Ry(A y + «I)- 2 Rj| 

|Kx + Ry(Ay + «I)~ 1 Q^Qy(Ay + a^gjj 
|F a (D) + Ry(A y + aI)- 1 Q y Q y (Ay + aI)-!RT| 

(352) 

|Qy(Ay + aI)Ry K X Ry(Ay + aI)Qj + I| 



2 |Qy(Ay + aI)RyF Q (D)Ry(Ay + aI)Q 



Y 



II 



-2 log 

- nl°g 



IQyAyRyKxRyAyQy + 1| 



QyAyR^D- 1 - HyHy) -1 RyAyQy + I| 
iHyK^Hy + 1| 



HyfD 



HyHy)" 1 ^ +I| 



(353) 
(354) 

(355) 



where (13501) is due to the continuity of the determinant in positive semi-definite matrices 
and (13481) . (13511) comes from the definition of Sy„, (13541) comes from the continuity of the 
determinant in positive semi-definite matrices and ( 13481) . and ( 13551) is obtained by using the 
singular value decomposition of Hy. Hence, (13491) and (13551) imply that any rate R inside 
the region lim^o T^-a(D) satisfies 



R > - lo^ 

~ 2 e 



IK 



X\Y\ 



IDI 



5 tog l7D=r 



IK 



X 



HyHy)" 1 ! 



nl°g 



|HyK X Hy +I| 



Hy(D 



HyHy^Hy 



(356) 
(357) 



Following a similar analysis, the limit of the information leakage in ( I346j) can be found as 



min - log 

0<^x\v<Vix\u^x 2 
Kxiv^D-'-HfHy)- 1 



IK 



X 



Kxivl 



"2 log 



HyKx|^Hy + 1| 
HyK X |yHy + I| 



+ -log|HyK X | C/ Hy + I| (358) 



which implies that any information leakage I e inside the region lim^o 7Z a (D) should be 
larger than (13581) ; completing the proof of Theorem [6) 
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N Proof of Lemma 1221 



We first prove the following lemma which will be used subsequently. 

Lemma 23 Let K(a) = (A + /(a)B)- 1 , < a < a*, where A y f(a)B hO, < a < a* 
and f{ot) is continuous in a. Then, we have 



limK(a) = (A + /(0)B) _1 



(359) 



Proof: In the proof of this lemma, we use the fact that if lim^oo C n = 0, we have 

oo 

(I + C)- 1 = ^(-l) n C n (360) 

n=0 

where C° = I [2U page 19]. Now, we consider 



K(a) = (A + /(«)B)- 1 

= K- l '\l + /(a)A" 1/2 BA" 1/2 ) _1 A~ 1/2 

where due to A y /(a;)B y 0, we have I y /(a)A _1 / 2 BA -1 / 2 y which implies 

lim (/(ajA-^BA-^j^O 

n— too 

Hence, we can use (13601) in (13621) to get 



K(a) = A' 1 ' 2 



5^(-l) n / n (a)(A- 1 / 2 BA- 1 / 2 ) n 



n=0 



A -i/2 



which implies 



lim K(a) = lim A~ 1/2 



a->0 



A -i/2 



^(-^/"(^(A^BA- 1 



/2Nn 



n=0 



A -l/2 



^(-^/"(OKA-'^BA- 1 



/2Nn 



n=0 



A -i/2 



A -V2 [I + Z^A-^BA^^A- 1 / 2 
(A + /(0)B)-' 



(361) 
(362) 



(363) 



(364) 



(365) 

(366) 

(367) 
(368) 



where (13671) comes from (I360p ; completing the proof of Lemma 
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We now consider (I347[) in Lemma [22] as follows 



K x ^ a = K X (K X + Ey^Ey^ (369) 

= (K, 1 + E-^)" 1 (370) 

= [K^ + Ry(A y + aI) 2 Ry] _1 (371) 

= [K^ 1 + RyAyRy + Ry(2aA y + a 2 I)Ry] _1 (372) 

where < a < a*. Equation f!369j) comes from (11751) . (13711) is due to the definition of Ey a . 
We note that K^ 1 + RyAyRy >~ 0, and thus, a* can be selected to ensure that 

K x x + RyAyRy >- Ry(2aA y + a 2 I)Rj (373) 

for all < a < a*. Hence, we can use Lemma [231 in f!372j) to get 

lim K x] ^ a = [K^ 1 + RyAyRy] _1 (374) 

= [K x x + RyAyQyQyAyRy] _1 (375) 

= (K x 1 + H^H y )- 1 (376) 

= K x ,y (377) 

where (13761) comes from the singular value decomposition of Hy and (13771) is due to (12 17ft ; 
completing the proof of (I347p . 

Next, we consider ( 13481) in Lemma [22] as follows 

F a (D) = £y a (£y )Q , — D) £y iQ — £y a (378 

= Ey Q (£y Q - D) X D (379 

= (D- 1 -Ey 1 J- 1 (380 

= (D _1 — Ry(Ay + aI) 2 Ry) _1 (381 

= [D" 1 - RyAyRy - Ry (2aA y + a 2 I)Ry] -1 (382 

= [D" 1 - RyAyQyQyAyRy - Ry(2aAy + a 2 I)Ry] _1 (383 

= [D^ 1 - HyHy - Ry(2aA y + a 2 I)Ry] -1 (384 

where < a < a*. Equation (138 ip comes from the definition of Sy Q and (I384j) is obtained 
by using the singular value decomposition of Hy. We note that D 1 — HyHy is strictly 
positive definite as (13381) indicates, and hence, there exists an a* such that 

D 1 — HyHy y Ry(2aAy + a 2 I)Rj (385) 



49 



for all < a < a*. Consequently, we can use Lemma [23] in (I384p to get 

lim FJD) = (D- 1 - HyHy) 1 (386) 
which completes the proof of Lemma 
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